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Abstract 

When climate forecasts are highly uncertain, the optimal mean squared error strategy is to ignore 
them. When climate forecasts are highly certain, the optimal mean squared error strategy is to use 
them as is. In between these two extremes there are climate forecasts with an intermediate level of 
uncertainty for which the optimal mean squared error strategy is to make a compromise forecast. We 
present two new methods for making such compromise forecasts, and show, using simulations, that 
they improve on previously published methods. 

1 Introduction 

Forecasts for changes in climate vary greatly in terms of the ratio of the predicted signal to the estimated 
uncertainty around that signal. For instance, numerical model derived predictions of changes in global 
temperature show a large ratio of signal to uncertainty, while forecasts of changes in local rainfall from 
the same models show a much smaller ratio of signal to uncertainty. Forecasts with such different levels of 
uncertainty should be used very differently. There are many ways one might model this mathematically, 
but one of the simplest is to consider the goal of the forecast to be minimizing the estimated mean 
squared error (MSE) of the final prediction. Such a framework leads one to ignore forecasts with a low 
value for the ratio between forecast signal and forecast uncertainty, and to use forecasts with a high value 
for this ratio, fntcrestingiy, there is then a grey area in between, where a compromise forecast can be 
produced that achieves a lower mean squared error than either ignoring the forecast or using it in full. 
These compromise forecasts make better use of the information in climate model output since they use 
this information to improve the forecast in parameter ranges where normally the model output would 
have to be ignored because of the high level of uncertainty. The statistical estimators used to make such 
co mpromise forecasts are somet imes known as minimum mean squared error estimators. 
In Jewson and Hawkind (|2009bh . we derived a simple method for making such a compromise forecast, and 



applied it to UK precipitation. We called the method a 'damped' forecast. In this article we present two 
new methods for making damped forecasts based on new minimum mean square error estimators for the 
mean of the normal distribution. We show, using simula t ions, th at these new methods both outperform 
the simple damping method from Jewson and Hawkind ( 2009b[ ) within the most important parameter 



range. 

In section [2] we describe the mathematical set up we will use, and review the mean squared error perfor- 
m ance of ignoring or us i ng a cl imate forecast. In section [3] we then describe the simple estimator used 
Jewson and Hawkind ( 2009b[ ). and its theoretical performance. In section [H we use numerical meth- 



ods to estimate the actual performance of the mean squared error estimator from Jewson and Hawkind 



( 2009b[) . In section [H we describe two new estimators, and compare the performance of all five methods 



discussed. In section [6] we summarise our results and discuss future directions. 



MSE performance of using or ignoring an uncertain climate 
forecast 



Our mathematical setup follows that of iJewson and Hawkind (|2009bh . We consider values of a future 
climate variable to consist of a contribution from current climate, a contribution due to a change in 
climate, and noise. We write this as: 

y = c + d + e (1) 



We will assume, without loss of generality, that c = 0, and so: 



y = d + e (2) 

Rather than deal with actual observed climate (which includes the noise) we simplify the algebra by 
considering future mean climate only, which we write as /j.. Then we have: 

ti = d (3) 

We imagine taking an estimate of the change in mean climate from a climate model (or climate model 
ensemble). We write the estimated change as d, and we assume that this is unbiased {E{d) = d). We 
make an estimate of the current climate c, and we assume that that is also unbiased. 
An unbiased prediction of future mean climate is then given by: 

fi = c + d (4) 

We then assume that c = 0. This assumption does lose some generality, in that terms in the MSE of the 
estimate c drop out of the subsequent expressions for total MSE. Essentially we are assuming that current 
climate can be estimated perfectly, which is clearly not true. However, since we are only interested, in 
this article, in the performance of predictions of the change in climate, this assumption does not change 
any of our results vis-a-vis the relative performance of different methods for predicting that change (and 
it simplifies the algebra a bit). 
We then have: 

fi = d (5) 
The mean squared error for this prediction of future mean climate is given by 

MSEi = E{il - ^if = E{d - df = E{d - E{d)f = V (6) 

where V — V{d) is the variance of the estimate of the change in climate. We don't discuss here how V 
might be determined, but it coul d, for instance, be der i ved fro m the spread of a climate model ensemble 
using the method we describe in ljewson and Hawkins! ( 2009a[ ). 



If instead of using climate model output in this way we decide to ignore it then our prediction is: 

A - (7) 

The mean squared error for this prediction is: 

MSE2 = E{[i - = E{0 - df = d^ (8) 

We now consider the mean squared error normalised by V, which we write as NMSE (and NRMSE for 

V ■ 



the RMSE version) . We also define the ratio = tt- For the two forecasts considered so far the NRMSE 



is: 

NRMSEi = 1 (using the climate model output) (9) 

and 

NRMSE2 ~ y ~ (ignoring the chmate model output) (10) 

Figured] shows the variation of NRMSE for these two forecasts versus r. We see that for large values of r 
{r > 1) the option to use the forecast has the lower mean squared error, while for small values of r (r < 1) 
the choice of ignoring the forecast has the lower mean squared error. This implies that for r < 1 the 
climate model output should be ignored. Among other benefits, the damping methods described below 
give a way of using climate model output to improve forecasts even when r < 1. 



3 A Simple Damped Estimator 

In[j ewson and Hawkins! (|2009b! ) we consider adjusting the prediction given in equation [5] using a damped 
prediction: 

fi = kd (11) 

for k between zero and one. 



We then derive the MSE optimal value of k, which is: 



The MSE for the prediction given by this optimal value of k is 

MSE3 = i?(A-M)'-i?(fcd-rf)2 = -^ (13) 

(we've skipped a lot of details of these derivations, since the details are given in Jewson and HawkinsI 
( 2009bM 



The normalised RMSE is then given by: 



NRMSE3 = y^ = y3: (14) 
This is shown, along with NRMSEi, and NRMSE2, in figured 

We see that in theory the damped prediction is always better than the simpler predictions based on either 
ignoring the model output or using it in full. For very small values of r it has more or less the same 
performance as ignoring the forecast, for very large values of r it has more or less the same performance 
as using the forecast, and for intermediate values of r « 1 is performs markedly better than either. 
However, there is a catch: the optimal value of k is unknown, since the expression for k given by 
equation [T^l depends on d and V, which are both unknown. For estimated values of k the mean squared 
error performance is likely to be significantly worse than the theoretical ideal. The performance of real 
estimates of k can be determined using numerical methods, and that is described in the next section. 

4 MSE performance of the simple damped forecast 

In[j ewson and HawkinsI (l2009bh we used a plug-in estimator for k. That is: 

k = ^ (15) 

d^ + V 

We now use numerical methods to determine how well this plug-in estimator performs, for a sample size 
of n = 10. Our methods work as follows: 

• For values of r between and 4 (with a step of 0.01), we simulate 1000 samples, each of size 10 
(each sample can be considered as an ensemble of climate model outputs) 

• For each sample, we apply the three prediction methods described so far (ignore the sample, use 
the sample in full, or use the damped prediction) 

• For each value of r we calculate the NRMSE for the resulting predictions 
The results are shown in figure [31 We see that: 

• For very small values of r the damped prediction performs less well than ignoring the forecast, but 
better than using the forecast in full 

• For large values of r the damped prediction performs less well than using the forecast in full, but 
much better than ignoring the forecast 

• For intermediate values of r, around r w 1, the damped prediction performs better than either of 
the simpler forecasts. 

We see that use of this damped forecast is not a panacea: it does not dominate the simpler methods for 
all values of r. However, for certain values of r it does improve the forecast. If one were forced to use 
just one of the three methods discussed one would probably choose the damped forecast since it never 
performs too badly, unlike ignoring the forecast which performs badly when the real signal is large, and 
unlike using the model output in full, which performs badly when the real signal is small. 



5 Two new minimum mean square estimators 



We now move to the main topic of this paper, which is to present two new minimum mean square 
estimators. The new estimators are based on the observation that the expression: 



is non-hnear in d and V , and so using a distribution of values for d and V may do better than using 
simple plug-in estimators for each. 

Motivated by Bayesian statistics, we therefore propose the following ad-hoc Bayesian estimator for k: 



where p{d, V\x) is the posterior probability for d and V. In other words, we consider all possible pairs of 
values of d and V, we calculate the damping for each pair, and we average all the damping coefficients 
together using a weighted average with the posterior probability as weights. 

Our second estimator goes a step further and models the prediction directly, without going through an 
intermediate fc, as: 



We extend our numerical tests to include the performance of these two new estimators, as follows: 

• To determine the posterior, we use the standard uninformative prior for the normal distribution 
(which is also the Jeffreys' Prior). 

• We evaluate the integrals using numerical integration. We discretize each dimension into 101 equal 
steps, over a range between minus and plus four standard errors. 

Figure [J] shows the MSE performance of the first of these two estimators. We see that it does better 
than the simple damping estimator up to just below r — 2, but less well for larger values of r. Over the 
range where the simple damping estimator beats the two simplest predictions, the new damping estimator 
clearly beats the simple damping estimator. 

Figure O shows the MSE performance of the second of the two new estimators. The second of the new 
estimators beats the simple damping estimator everywhere except for very small r. Over the range 
where the simple damping estimator beats the two simplest predictions, the second of the new damping 
estimators clearly beats the simple damping estimator. However, the relationship between the two new 
estimators is complex. The second beats the first for larger values of r, and vice versa. 
Overall, there are now four estimators which are the best, depending on the value of r: 

• For very small r, completely ignoring the forecast is best (for r < 0.74). 

• For slightly larger r, the first of the new damping estimators is best (for 0.73 < r < 1.19). 

• For r larger again, the second of the new damping estimators is best (for 1.18 < r < 2.03) 

• And finally, for large r, using the forecast in full is best (for r > 2.02). 

The simple damping estimator is never the best, although it is not totally dominated by any of the other 
estimators. 

6 Summary 

We have discussed the mean squared error performance of forecasts derived from the output of numerical 
climate models. We have considered the mean squared error performance of 5 types of forecast derived 
from numerical climate model output: 

• Using the forecast as is 




(17) 




(18) 



• Ignoring the forecast 

• Damping the forecast using the damping scheme of 



Jewson and Hawkins! ( 2009bf ) 



• Damping the forecast using a new damping scheme 



• Damping the forecast using a second new damping scheme 
We hav e found that the new damping schemes outperform the original damping scheme of Jewson and Hawkins! 



(|2009bll over the most important range of parameter values (which is the range over which the simple 
damping scheme beats ignoring the model output and using it in full). Within this range the two new 
schemes each perform best for different ranges of the parameters. 

The obvious next question is: how should one choose which of these methods to use in practice, given 
real climate model output? Strictly speaking, this is not answered by our results, since we show the MSE 
versus an unknown parameter (the ratio of the unknown signal size to the unknown uncertainty around 
the signal). One could decide which method to use by replacing the unknown parameter by an observed 
estimate, but this would be to ignore uncertainty on that observed estimate. There may, therefore, be a 
better way to decide which method to use and when. We are looking into it. 
There are also many other questions that arise from this work, such as: 

• Whether it would be worth considering metrics other than MSE 

• Whether there are damping methods that work better than those proposed here 

• Which climate forecasts, for what variables and at what lead times, fall into the various ranges of 
the parameter r. 
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Figure 1: The normalised RMSE performance of a) using a climate forecast (horizontal black line) and 
b) ignoring a climate forecast (diagonal black line), versus the ratio of signal to uncertainty. 




Figure 3: As figure 1, but now including the RMSE performance of the plug-in estimator for the damping 
model (red line). 




Figure 5: As figure 3, but now including the RMSE performance of the second new estimator (blue line). 



